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DUAL BUFFER VIDEO DISPLAY SYSTEM FOR 
THE DISPLAY OF ASYNCHRONOUS IRREGULAR 
FRAME RATE VIDEO DATA 

5 . 

BACKGROUND OF THE INVENTION 
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la computer-based video coxxnaimicatioQ systems, a 
video signal is obtained from the camera at a constant 
frame rate but, after transmission across the asyncbro- 40 
nous or non-ideal network, the frames arrive at irregu- 
lar intervals. Some frames arrive early, some are de- 
layed, and bunching can occur. The display device at 
the receiving terminal, however, generally requires a 
constant frame rate supplied to it (e.g-» to match the 45 
raster scan rate of a CRT). In such systems it is there- 
fore necessary to match the irregular arrival of frames 
over the network with the constant supply required to 
the output screen, 

50 



55 



. The drsagnrr of 
computer based video coonnuiiicarion systems is there- 60 
fore faced with the problem of how to achieve regular 
play-out of the asynchronous in coming video signal 
while, at the same time, ™^^g the number of buff- 
ered video frames. 
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- nominal arrival time can be properly displayed. Only if 
a frame arrives more than T(L) late, will the AVK and 
circular buffer empty and the video image will freeze. 
To decrease the risk of buffer starvation, the buffer size 
5 can be increased to make T(L) larger, but with a 15 
frames per* second transmission rate, storing only 10 
frames adds a delay of i second. If the effectiveness of 
interactive applications such as video conferencing is 
not to be seriously degraded, only a handful of frames 
10 can be buffered with T(L) correspondingly small. 

The control process is responsible first for receiving 
data into the circular buffer, and then for forwarding it 
to the AVK. There is no control over output from the 
AVK, which is at a filed rate. As explained in more 
15 detail below, the AVK requests frames from the circu- 
lar buffer as required Clearly, if frames are present in 
the circular buffer, then these can be forwarded to the 
AVK. However, in video conferendng or other tnter- 
FIG. 1 shows a network 2 of computers 4, 6, 8 con- active applications where the overall amount of buffer- 
nected by an asynchronous communication channel 15 20 ing is timffd, there may occasionally be particularly 
(eg., LAN or ISDN)- A camera 16 at a first computer long delays on the network during which time the cir- 
4 obtains a video signal, normally of the user, which is cular buffer empties. In this case, the control process 
compressed and submitted to the network. The signal is reacts by loading the AVK with null frames. A null 
then transmitted down the communication channel in frame is essentially the same as the preceding frame, so 
packet format before arriving at the destination com- 25 that, as far as . the viewer is concerned, video image 
puter 6. Typically, this second computer includes hard- temporarily freezes. Thus, each time the control process 
ware such as die Intel/IBM ActionMedia H (AMU) fails to find frames in the circular buffer, the requisite 
card, which is responsible for actually decompressing number of null frames are loaded into the AVK instead, 
and displaying the video image on the screen 9. In video Although the user may not notice the insertion of 
cOTiferencing applications, the reverse process also oc- 30 individual null frames, each null frame adds to the over- 
curs; Lc, the second computer is simultaneously send- all delay in the system (Lc_ it is effectively another form 
ing an image of its user, back to the first computer 4 for of buffering). If more and more null frames are inserted 
display. It is also possible to set up multi-way confer- into the video stream, then this will, again, lead, to an 
ences, - intrusive delay between transmission and display. This 

With reference now to FIG. 2, the mcoming video 35 problem can be overcome. by the circular buffer throw- 
signal, from the communication subsystem 15 arrives at ing away real data when the delayed frames do finally 
the workstation 13 for display on the associated monitor arrive. These frames are then effectively lost, allowing 
9. The signal is transferred first to a buffer 23, and then the displayed image to catch up with the mcoming 
to the AMU card 125 or, more particularly, to the Au- signal It is the presence of two buffers that gives the 
dio Visual Kernel (AVK) mterface buffer 25 of the 40 flexibility to lose frames in this way, and so cope with 
AMU card. . The buffer 23 provides a FIFO queue, occasional delays longer than T(L). 
conveniently implemented as a areolar buffer. A con- The technique used to discard frames exploits the face 
trol process 27 is responsible first for reading mcoming that, due to the IfrmtiM bandwidth of the channel, the 
data into the circular buffer, and then for transferring video signal is compressed before transmission over a 
data from the circular buffer to the AVK. 45 computer based cxraimunication line. Basically, two 

Video images arc captured at the source computer at types of compression, spatial and temporal, are used la 
a frame rate of 15 frames per second (in tins particular the former, the redundancy within a single frame is 
embodiment), which is sufficient to provide moderate removed, for example, by using the fact that adjacent 
quality video. This is also the rate at which they are pixels often have closely related brightness and color 
read out of the AVK to the screen. However, the trans- 50 values. A frame encoded using only spatial compression 
mission rate over the network is variable, depending on is known as a "still frame". Temporal co mpre ssion 
network load, etc, so that the arrival, rate at the end of achieves a further level of compression by exploiting 
the computer subsystem departs from this 15 Hz clock. the fact that the lunxmosity and color, of the same pixel 
Changes in CPU activity at the source and destination in two consecutive frames are, again, likely to be highly 
computers can also lead to variations in the effective 55 correlated. Therefore, in temporal compression, a frame 
frame arrival rate. Individual frames can have either a is encoded as a "relative frame" in terms of hs differ- 
positive or negative offset from their nominal arrival ence from the previous frame (we assume that a relative 
time, although it is assumed that frames do, in fact, frame is also spatially compressed). The greatest reduc- 
arrive in the coxiect sequence. It should be noted that ticn in data is achieved if every frame (apart from the 
the variation in arrival times is such that, even if the 60 first) is a relative frame, but this is highly error prone 
hardware could display each frame directly on arrival, since the loss.of a single frame will produce defects that 
the resulting sequence would be so temporally distorted persist for all subsequent frames.. There f ore, as a com- 
as to be unwatchable. promise, every Nth frame can be sent as a still frame, 
essential. with all intervening frames as relative frames, so that 
Together, the AVK and. circular buffer compensate 65 the result of compression is a regularly spaced series of 
for the variable arrival rate of the video frames by intro- frames whose size varies somewhat according to the 
ducing a time-lag, T(L). between the received and dis- temporal and spatial content of the data and, of course, 
played images. Any frame arriving within T(L) of its whether that particular frame is a still or relative frame. 

.-. -4- 
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When the buffer is not full, then incoming frames 
can be added to the buffer in the normal way. However, 
when the buffer is full, there are two possible actions- If 10 
the incoming frame is a stxH fr» m ^ then the entire buffer 
is flushed before the mcommg sail flame is added to the 
queue. Alternatively, if the incoming frame is a relative 
frame, then only relative frames below (Lc~, that arrived 
earlier than) a still frame are fhwfrpd. This is because the L5 
previous still frame is still required to make sense of the 
relative frwnw In either case, flushing the buffer results 
in some frames being thrown away, and so the displayed 
image catches- up slightly with the received image. 



25 



30 



Turning now to the AVK, frames are read out from 
the AVK for display at a fixed rate. This Leads to the 
possibility of buffer starvation if the AVK contains no 
more frames to read out to the screen. In such an even- 35 
tuality, the AVK pipeline needs to be reset, requiring a 
considerable system overhead during which time the 
video image is not updated, in contrast to the circular 
buffer, which can be emptied and refilled without pen- 
alty. 40 

Accordingly, a lower limit, V(L), is set for the num- 
ber of frames in the AVK This value is selected to 
substantially preclude buffer starvation yet, at the same 
time, not introduce an unacceptable delay. The control 
process responsible for transferring frames from the 45 
circular buffer to the AVK then tries to roamfaTn the 
number of frames in the AVK as close as possible to but 
slitthilv above V(L). . 
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Once the control process has determined the number 
60 of frames to transfer to the AVK. it can either send this 
as a single request or as an appropriate number of re- 
quests for mdrvidual frames. In the latter case, the circu- 
lar buffer can respond simply to each request by trans- 
ferring a frame if available, or inserting a null frame if 
65 not 
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